Communication device and method for deaf and mute persons

ABSTRACT

A communication device for deaf, hearing impaired, or mute persons comprises a processor control system which makes use of a microphone and a speech recognizer to receive and process audio data (speech or non-speech) to determine whether or not a dangerous situation exits within the environment surrounding the user. The system comprises a key pad and/or stylus and tablet information input system to accommodate communication from the user to the persons in the surrounding environments and a visual display capability to transmit the information so acquired by way of a projection apparatus in the form of characters and in the language to which the user is accustomed. Indicator signals which correspond to dangerous or cautionary situations relating to abnormally loud noises, or readily recognized sound patterns, such as a siren may also be displayed to the user, as may be information related geographic location, distance to a preset destination, or other personally useful information.

This application claims the benefit of U.S. Provisional Application Nos.60/026,400, and 60/025,480, filed Aug. 29, 1996.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates generally to a device for communication betweenpersons who are deaf, hearing impaired, or mute. More particularly, theinvention is directed toward a communication system for deaf, hearingimpaired or mute persons which allows the user to visualize speech andother sounds directed at him through various audio sources.

2. Background of the Invention

Historically, deaf, hearing impaired, and mute persons have had adifficult time communicating, both among themselves and with others notsimilarly handicapped. In fact, it is often more difficult to conduct asimple conversation between a non-handicapped person and one that isdeaf or mute, because the non-handicapped person usually is not familiarwith American Sign Language (ASL).

Even when the persons communicating are familiar with ASL, sign languageis not a very convenient method of communication when a large group isinvolved. Expensive television cameras and video screens must beemployed to transmit messages to large audiences. That is, unless thereceiver of the message is so close to the message transmitter that heor she can distinguish ASL gestures and the expressions on the messagetransmitter's face, communication is simply not possible. As mentionedpreviously, if the message receiver has not been trained to understandASL, then communication is also not possible.

Further, deaf persons often cannot enjoy television or the theaterbecause the actors are not visible, or have turned their backs to thefront of the stage so that lip-reading is impossible. Of course, thisdifficulty is also encountered during day-to-day communication scenarioswhenever the speaker or communicator is not facing the deaf person, oris not located in the visual line-of-sight range of the deaf person. Inaddition, there are occasions when an unusually loud noise is used toalert persons in the vicinity that a dangerous or cautionary event isabout to occur. This can be the siren of an ambulance or fire engine, anexplosion, the horn of an automobile or truck, etc. While a person ableto hear such sounds can immediately identify these circumstances asrequiring caution or flight, the deaf person will be unaware that he isin danger.

Several elements of the instant invention have only recently becomegenerally available due to the general trend of technologyminiaturization and a reduction in the price of sophisticatedmicroprocessor control systems. More specifically, this includes thetechnologies of speech recognition, short-range infrared and radio datacommunication, personal video displays, and handwriting recognition. Thepresent invention is directed toward overcoming the communicationdifficulties set forth above.

It is desirable to have an apparatus and method enabling communicationsbetween deaf, hearing impaired, or mute persons and others, whethersimilarly handicapped or not. It is also desirable to have an apparatusand method for use by deaf or hearing impaired persons to enable them toenjoy television or movie theaters without subtitles, ASL interpreters,etc. In addition, it is desirable to have an apparatus and method whichenables deaf and/or mute persons to communicate with others who may notbe in close proximity. Furthermore, it is desirable to have an apparatusand method which enables communication between a single individual and agroup, whether or not all individuals participating in the communicationhave the ability to speak or hear without impairment.

While ASL has been developed for enhancing the communication abilitiesof deaf and mute people, most non-handicapped persons are not trained inits use. Even those that are trained are unable to use it forcommunication in circumstances where line-of-sight communication isimpossible or impractical. Thus, there exists a long-felt and widespreadneed to provide alternative communication apparatus and methods fordeaf, hearing impaired, and mute persons which can be used at adistance, in situations where line-of-sight communication is impossibleor impractical, where communication with a group of mixednon-handicapped persons and deaf and/or mute persons is desired, or ineveryday situations which render common means of understanding byhearing impaired persons (e.g., lip-reading) ineffective. This isespecially the case for dangerous situations in which a loud soundemanates from an undisclosed location and the deaf person remainsunaware of its existence. The apparatus and method of the presentinvention, discussed in greater detail below, clearly satisfy this need.

SUMMARY OF THE INVENTION

The present invention provides a communication system for deaf, hearingimpaired, and mute persons, or those communicating with hearing impairedand/or mute persons. The communication system of the present inventioncomprises a message reception means to receive audio information, suchas a microphone and an optional filter; an information conversion means,such as a speech recognizer, to convert the received information intospeech data; a data processing means, such as a microprocessor, toorganize the speech data into an appropriate and meaningful symbolicrepresentation; and a display means, such as a see-through liquidcrystal display to display the symbolic representation of the speechdata (usually English language text symbols) to the user. The processingmeans also has the ability to process other audio data (e.g. non-speech)to determine whether or not a dangerous situation exists within theenvironment surrounding the user.

Further, the apparatus and method of the present invention comprises amessage entry means, such as a keypad and/or stylus-and-tablet toacquire messages from the user. The apparatus and method of the presentinvention also comprises a message transmission means, such as a voicesynthesizer and speaker, to convert the user messages into message data,and convey the message data as audio output, preferably speech, to theuser's immediate physical environment.

In another aspect of the present invention, a method and apparatus isprovided for communication between hearing impaired, deaf and/or mutepersons and others, including a group of people, which involves use ofan optional element of the message transmission means, comprising aremote data transceiver electrically connected to the processor of thepresent invention so as to transmit and receive message information fromother communication systems that are not in the immediate vicinity.

Other features of the apparatus and method include the ability toproduce indicator signals for display to the user which correspond todangerous or cautionary situations relating to abnormally loud noises,or readily recognized sound patterns, such as a siren. The apparatus mayalso comprise a cellular phone which operates as an adjunct to themicrophone and speaker elements, providing mobile telephonecommunications for the handicapped. In addition, indications can betransmitted to the user representing information related to geographiclocation, distance to a present destination, or other personally usefulinformation. The above and other advantages of this invention willbecome apparent from the following more detailed description, inconjunction with the accompanying drawings and illustrative embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of the present invention indicating auser's perception of a projected message image.

FIG. 2 is a stylized representation of the display for the user,including messages and various information indicators.

FIG. 3 is a simplified block diagram of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1 illustrates a perspective view of an exemplary embodiment of thepresent invention, as it is worn by user 200. A communication system 10for deaf, hearing impaired, or mute persons optionally comprisesheadband 120, or other suspension means, such as eyeglasses, goggles, ora helmet, for supporting the various component elements which comprisethe present invention.

The user 200 receives messages, including speech and other audioinformation, by way of a message reception means, such as a microphone70, most preferably a self-powered and unobtrusive clip-on type. Thecommunication system 10 also comprises a data processing means, such asa wearable processor 20, which may be co-located with the user 200 andelectrically connected to a display means, such as a projector 50, so asto control the content of the projector display 140.

A keypad-tablet 35, or other user message entry means, comprising akeyboard 40 and a handwriting recognition display 42, is manipulated bythe user 200 to present user message information to the wearableprocessor 20 for transmission to the environment surrounding the user200 by way of a speaker 60 and/or a remote data transceiver 72. A stylus45 can be used to write on the recognition display 42 so that theprinted or handwritten message entry by the user 200 can be recognizedby the wearable processor 20 and communicated to the environmentsurrounding the user 200. A keypad-tablet 35 may be connected directlyto the processor 20, or communicate with the processor 20 via radio orother electromagnetic means, such as by infra-red signals (commonlyfound on television remote control devices). The wearable microprocessor20 is capable of transmitting messages into the environment by way of amessage transmission means comprising a synthesizer 57, which convertsuser messages into message speech data, and a speaker 60, which presentsaudio information to the environment surrounding the user 200.

Turning now to FIG. 2, a representative sample of the display of thecommunication system 10 can be seen. Essential to the practice of theinvention is the message area 100, which contains the collection ofalphanumeric characters which communicate speech to the user, after ithas been recognized from audio information in the surroundingenvironment, or submitted to the processor 20 by way of a remotelytransmitted message from a similar device. It is preferred that themessage area 100 be located on the periphery of the display 140 so thatthe viewing area directly in front of the user 200 remains clear.However, it is most preferred that projector 50 presents a display 140to the user 200 which can be seen through (a “see-through” display), sothat any message text or indicators shown thereon will not interferewith the user's normal line of sight. While shown as a single messageline display, those skilled in the art will realize that several messagelines can also be projected onto display 140. Most preferably, messagearea 100 will comprise two lines of material presented in teleprompterfashion.

A first indicator 80 and a second indicator 90 represent optionalelements of the display 140. The first indicator 80 may consist of analarm icon, flashing light, or some type of analog display symbol whichindicates the amount of noise in the environment. Of course, thesepossibilities are listed by way of example, and not by way oflimitation. Similarly, the second indicator 90 may represent ageographical position location, generated by an attached GlobalPositioning Satellite (GPS) system, the distance of the user from aselected location, or the sound level of the surrounding environment indecibels. Of course, these specific examples of the content of secondindicator 90 are not meant to limit the possibility for display ofinformation to the user. Any information which can be processed by thewearable processor 20, either acquired from the surrounding environment,or pre-programmed into the unit, can be displayed to the user 200 at anappropriate time. Similarly, the time of day or of a specific event canalso be included in the content of the second indicator 90.

Turning now to FIG. 3, a simplified block diagram of the communicationsystem 10 can be seen. A power source 30, which may consist of arechargeable battery, solar cell, or other electric current powergenerator, provides power to the wearable processor 20. To processsurrounding environmental audio data, the environmental messagereception means comprising the microphone 70, and optionally, the filter150, is used to acquire audio information. A filter 150 is mostpreferably implemented as a band-pass filter which passes frequenciesfrom about 500 Hz to about 2,999 Hz. As a further aid to recognizingspeech, any of several noise cancellation techniques can be implementedby the wearable processor 20 or the speech recognizer 55. Informationconversion means, such as a speech recognizer 55, preferably operates ina manner that will accept continuous speech input and other audioinformation (such as a police siren tone pattern or abnormally loudsound), using templates and a vocabulary lookup table, or other methodswell known in the art, and convert it into speech data. The dataprocessing means processor 20 will then organize the speech data into ameaningful symbolic representation for transmission to the projector 50.

To process user initiated messages, the message entry means such as thekeypad-tablet 35 is used to acquire user message information, which ispassed on to a message transmission means, comprising a speechsynthesizer 57 and a speaker 64, producing an audible signal in theuser's immediate vicinity. Synthesized speech, produced by any ofseveral methods well known in the art, is most preferably formed by thesynthesizer 57 so as to mimic the gender of the user. In this manner, amute person can communicate with those that can hear. The messagetransmission means may alternatively comprise the synthesizer 57 and aremote data transceiver 72, wherein the keypad-tablet 35 user inputmessage information can alternatively be sent to a remote location viathe remote data transceiver 72 and an energy transceiver 77. The remotedata transceiver 72 comprises an electromagnetic energy transceivercapable of receiving and transmitting electromagnetic energy using anappropriate energy transceiver 77. If the electromagnetic energyconsists of infrared or other light, then a laser diode transmitter andreceiver, or similar devices, would be most appropriate to use as theenergy transceiver 77. If the electromagnetic energy consists of radiowaves, then an antenna designed for the desired frequency and bandwidthshould be used as the energy transceiver 77.

By means of the above-described apparatus and method of communication,any person, whether mute or not, can communicate at a distance withanother person who is hearing-impaired or deaf. Of course, suchcommunications can also exist between persons at each end of thecommunication link who are not disabled. In any event, those skilled inthe art will recognize that such an arrangement of remote radiocommunication devices also lends itself to multi-channel communication,which allows groups of disabled users to communicate amongst themselves,either out loud, or silently. Another example of system use might enableall English speakers to communicate on one channel, and all Spanishspeakers on another. Such multi-channel operation is desired to enhanceuser privacy and the practicality of the system. It is anticipated thata single “international” frequency, along with several “national”frequencies will be allocated to facilitate communications betweenpeople of differing countries, and the same country, as desired.

The processor 20 can be programmed to recognize a dangerous or acautionary situation, due to the sheer magnitude of the electricalsignal received, or based on a recognized pattern, and will thereafterdisplay an alarm signal by way of the first indicator 80 to the user 200via the projector 50 and the display 140. Operation of the communicationsystem 10 by using audio input and output (via the microphone 70 and thespeaker 60) obviates the need for ASL, and allows deaf, hearingimpaired, or mute persons to communicate with others that are similarlydisabled, and also with individuals having unimpaired hearing and speechcapability.

Environmental information, normally existing as local audio information,can also be received by way of the energy transceiver 77 and the remotedata transceiver 72. Such remote information will normally originatefrom another communication system 10 at a remote location (i.e. outsideof the line-of-sight, or out of normal hearing range) and consist ofaudio information, or speech data. The acquired signals can then beprocessed by the speech recognizer 55 or, if already processed by theremote communication system 10, the received speech data can be sentdirectly to the processor 20 for organization into an appropriatesymbolic representation of the data, and on to the projector 50 forviewing by the user 200 and, optionally, transmitted to the externalenvironment via the speech synthesizer 57 and the speaker 60.

An optional element of the communication system 10 is a storage device62, which can store several canned messages for recall by the user 200.This provides a convenient means of rapidly communicating standard wordsand/or phrases to the external environment. In addition, the storagedevice 62, which comprises battery-powered RAM memory, EPROMs, diskdrives, or tape drives, can be used to save messages that have beenreceived from the external environment for receipt and processing at alater time by the user 200.

In another embodiment of the communication system 10, a cellulartelephone (not shown) can be used as a supplement to the microphone 70and the speaker 60. When the telephone receives an incoming call, thefirst indicator 80 within the display 140 can be activated, along with amessage to the user 200 that the “Phone is ringing.” The user 200 canthen choose to answer the phone, and any input via the keypad-tablet 35will be converted to synthesized speech, which is sent to the telephonereceiver, instead of the speaker 60. Similarly, any voice emanating fromthe telephone speaker output can be directed to the filter 150 and thespeech recognizer 55 for conversion to text and display via theprojector 50. This embodiment provides mobile telephone operationcapability to deaf, hearing impaired, or mute persons, obviating theneed for teletype (TTY) devices on one or both ends of a telephoneconversation.

The processor 20 is understood to be equipped with the necessaryhardware and software to translate speech it receives from therecognizer into another language if desired, and then to display theresulting message via the projector 50 onto the display 140. Therecognizer 55 may also be implemented as part of the softwareprogramming for the processor 20, obviating the need for a separatehardware element. The invention also anticipates a combination of someor all of the separate elements, such as the microphone 70, the filter150, the recognizer 55, the processor 20, the speaker 60, thesynthesizer 57, and the storage device 62 into a single, integrated,hybrid unit. The separate elemental units are illustrated as such forclarity, and not by necessity. Likewise, the processor 20 can beremotely located to reduce local weight/power requirements. A separateradio data communications path (not shown) can be used to transmit andreceive processed audio and video data to and from the user 200, ifdesired.

The projector 50 may project the display 140 out into space in front ofthe user, as occurs with a holographic projection system, or may consistof any of several head-mounted optical projection systems, such as theHOPROS™ unit produced by Optronic Instruments & Products, the PROVIEW™system produced by KEO, or the Mark II GP Series of wearable displaysavailable from Seattle Sight Systems, Inc. The projector 50 may alsocomprise a see-through liquid crystal display with the display 140indicated directly thereon, or a retinal scanner, such as the VirtualRetinal Display being developed by the Human Interface TechnologyLaboratory at the University of Washington. In more sophisticatedembodiments of this invention, the display 140 as perceived by the user200 can be adjusted to accommodate vision correction or deficiency sothat any image perceived by a particular user 200 will appear in sharpfocus. The adjustment can be programmed into the processor 20 by sensinga user-controlled input (not shown), or mechanically adjusted by movingthe display 140 closer to, or farther away from, the eye. Therecognition display 42 and the stylus 45 input devices are similar tothose readily available in the market, and can be identical to thoseused on electronic appointment calendars or pen-top computers. Therecognition display 42 can be separated from the keyboard 40 and usedalone, without the use of the keyboard 40. In addition, the recognitiondisplay 42 may contain the intelligence necessary for independenthandwriting or printing recognition, or may pass on electrical signalsso that the wearable processor 20 can recognize the content of the usermessage entry by way of writing recognition algorithms well known in theart. The present invention anticipates that the user 200 will receivefeedback at the message area 100 in character text form for each messageentered on the recognition display 42. In this way, the user 200 can beassured that what is written on recognition the display 42, using thestylus 45, is properly translated into text for transmission to others.In an alternative embodiment of the present invention, transmission ofthe characters will not occur until the user has authorized suchtransmission.

The communication system 10 anticipates use as a real-time languagetranslator for disabled persons, as well as for those who can hear andspeak. Language translation software modules are readily available foruse with text received by the wearable processor after voice recognitionoccurs, or after messages have been input via the keypad-tablet 35.Several such language translation program modules may be simultaneouslyresident in the processor 20 memory. The user 200 can, for example,write or type a message in English, and after textual translation, itwill be “spoken” in a selected alternate language via the speechsynthesizer 67 and the speaker 60. Additionally, a foreign languagespeaker may verbally address the user 200, who, upon selecting theproper translation software module for execution by the processor 20,may then (after recognition and translation occurs) be enabled to readthe translated message at the message area 100 of the display 140 (orhear the synthesized version thereof) in the user's 200 own language.Deaf/mute users may also communicate between themselves, each in theirown different language, by a similar process.

The present invention will also facilitate round-table discussions amongthose conversing in two or more languages, by way of speech and/orremote transceiver broadcast and transmission. It is anticipated thatappropriate translation codes, either oral or electromagnetic, will besent with the text so that the communication system 10 can automaticallyengage the proper software translation module for use by the processor20. Such operational methods will obviate the need for manual selectionof translation software modules.

It is also anticipated that the user 200 will be able to enjoytelevision or theater using the communication system 10 by employing themicrophone 70 and the speech recognizer 55 to produce text via theprocessor 20 for presentation to the user 200 via the display 140. Byusing communication system 10, it does not matter whether the speaker isdirectly facing the listener or has turned away from the front of thestage. In addition, even if the speaker has left the immediate area ofthe listener, the communication system 10 will enable the listener toreceive the appropriate message in an environment in which audiocommunication is possible. At more remote locations, the user may employthe remote data transceiver 72 and the energy transceiver 77 to enablesuch communications. Such use of the system 10 will obviate the need forexpensive subtitle apparatus for use with televisions, and dependence onthe proper coding which much be supplied by the television networktherefor.

The method of communication for deaf, hearing impaired, or mute personspresented herein comprises the steps of: receiving audio information;converting the audio information, whether speech or other information,such as a siren or explosion, intospeech data; then converting thespeech data into a symbolic representation which is meaningful to theuser (e.g. German language text characters for a German languagespeaker); and displaying the symbolic representation to the user, eitherin the space in front of the eye, or directly onto the eye, as occurswith retinal scanning. The method further comprises acquiring usermessages which are intended for transmission to others in thesurrounding environment and transmitting those messages to thesurrounding environment in the form of audio information.

Although the invention has been described with reference to a specificembodiment, this description is not meant to be construed in a limitingsense. On the contrary, even though only specific devices have beenshown to be mounted to the headband 120, all elements of the instantinvention can be mounted thereon, given sufficient miniaturization.Also, various alternative stylized displays can be used, other than thatshown in FIG. 2. As long as the user is given visual indication of themessage he receives from the environment, the spirit of this inventionis effected. Other various modifications of the enclosed embodimentswill become apparent to those skilled in the art upon reference to thedescription of the invention. It is, therefore, contemplated that thefollowing claims will cover such modifications, alternatives, andequivalents that fall within the true spirit of the scope of theinvention.

We claim:
 1. A communication system for deaf, hearing impaired, or mutepersons, comprising: a message reception means for receiving audioinformation; an information conversion means to convert said audioinformation into speech data, said information conversion means being incommunication with said message reception means; a data processing meansfor organizing said speech data into a symbolic representation of saidspeech data, said data processing means being in communication with saidinformation conversion means; a visual display means for displaying saidsymbolic representation, said display means being in communication withsaid data processing means; a message entry means for acquisition ofuser messages, said message entry means being in communication with saiddata processing means; and a message transmission means for conversionof said user messages into message data and transmission of said messagedata into the surrounding environment, said message transmission meansbeing in communication with said data processing means.
 2. The system ofclaim 1 wherein said message reception means further comprises amicrophone.
 3. The system of claim 1 wherein said message receptionmeans further comprises a microphone and a filter.
 4. The system ofclaim 1 wherein said information conversion means comprises a speechrecognizer.
 5. The system of claim 1 wherein said information conversionmeans comprises a continuous speech recognizer.
 6. The system of claim 1wherein said data processing means comprises a personal computer.
 7. Thesystem of claim 1 wherein said data processing means comprises apersonal digital assistant.
 8. The system of claim 1 wherein saiddisplay means is affixed to a pair of eye glasses.
 9. The system ofclaim 1 wherein said display means is affixed to a headband.
 10. Thesystem of claim 1 wherein said display means comprises a liquid crystaldisplay.
 11. The system of claim 1 wherein said display means comprisesa see-through liquid crystal display.
 12. The system of claim 1 whereinsaid display means comprises a retinal scanner.
 13. The system of claim1 wherein said display means comprises a holographic projection system.14. The system of claim 1 further comprising a storage device, saidstorage device being in communication with said data processing means.15. The system of claim 1 further comprising a cellular telephone, saidcellular telephone being in communication with said informationconversion means and said message transmission means.
 16. The system ofclaim 1 further comprising a remote data transceiver, said transceiverbeing in communication with said information conversion means and saidmessage transmission means.
 17. A method of communication for deaf,hearing impaired, or mute persons, comprising the steps of: receivingaudio information; first converting said audio information into speechdata; second converting said speech data into a symbolic representationof said data; displaying said symbolic representation in a visualformat; acquiring user messages; and transmitting said messages to thesurrounding environment in the form of audio information.
 18. The methodof claim 17 wherein said receiving step is accomplished with amicrophone.
 19. The method of claim 17 wherein said receiving step isaccomplished with a microphone and a filter.
 20. The method of claim 17wherein said first converting step is accomplished with a speechrecognizer.
 21. The method of claim 17 wherein said second convertingstep is accomplished with a personal computer.
 22. The method of claim17 wherein said displaying step is accomplished with a liquid crystaldisplay.
 23. The method of claim 17 wherein said displaying step isaccomplished with a see-through liquid crystal display.
 24. The methodof claim 17 wherein said displaying step is accomplished with a retinalscanner.
 25. The method of claim 17 wherein said displaying step isaccomplished with a holographic projector.
 26. A communication systemfor deaf, hearing impaired, or mute persons, comprising: a messagereception means for receiving audio information; a data processing meansto convert said audio information into speech data and organize saidspeech data into a symbolic representation of said speech data, saiddata processing means being in communication with said message receptionmeans; a display means for displaying said symbolic representation, saiddisplay means being in communication with said data processing means; amessage entry means for acquisition of user messages, said message entrymeans being in communication with said data processing means; and amessage transmission means for conversion of said user messages intomessage data and transmission of said message data into the surroundingenvironment, said message transmission means being in communication withsaid data processing means.
 27. The system of claim 26 furthercomprising a remote data transceiver, said transceiver being incommunication with said data processing means and said messagetransmission means.